Aiming at the problem of low detection rate of depth feature based object detection method Faster R-CNN （Faster Region-based Convolutional Neural Network） in flame detection tasks, a color-guided anchoring strategy was proposed. In this strategy, a flame color model was designed to limit the generation of anchors, which means the flame color was used to limit the generation locations of the anchors, thereby reducing the number of initial anchors and improving the computational efficiency. To further improve the computational efficiency of the network, the masked convolution was used to replace the original convolution layer in the region proposal network. Experiments were conducted on BoWFire and Corsician datasets to verify the detection performance of the proposed method. The experimental results show that the proposed method improves detection speed by 10.1% compared to the original Faster R-CNN, has the F-measure of flame detection of 0.87 on BoWFire, and has the accuracy reached 99.33% on Corsician.The proposed method can improve the efficiency of flame detection and can accurately detect flames in images.